Skip to content

Add a sweeper for Terraform state locks orphaned by killed workers - #797

Closed
emmanuelmaduwuba-lyft wants to merge 2 commits into
mainfrom
terraform-lock-sweeper-pr
Closed

emmanuelmaduwuba-lyft wants to merge 2 commits into
mainfrom
terraform-lock-sweeper-pr

Conversation

@emmanuelmaduwuba-lyft

@emmanuelmaduwuba-lyft emmanuelmaduwuba-lyft commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem

A worker that is killed mid-run (SIGKILL after the 60s SIGTERM window, a pod kill, an OOM) leaves the Terraform state lock behind: the S3 <root>/terraform.tfstate.tflock object and the DynamoDB lock item. Every later plan for that root fails with 412 PreconditionFailed / ConditionalCheckFailedException until someone deletes both by hand. A recent rollout left locks on dozens of roots that sat for about 18 hours.

Change

  • Add server/neptune/lyft/locksweeper: a cron-scheduled Temporal workflow (every 30 minutes, fixed workflow ID so one schedule exists cluster-wide) that lists the locks in both stores and, for each one:
    • skips it if it is younger than min_age_minutes (default 120, config floor 60, the longest a Terraform activity runs) or if Temporal has an open Terraform workflow for the root. If the Temporal check fails the lock is skipped.
    • alerts (error log and a counter) without deleting if it is an apply lock, any other operation, or held by someone other than the Atlantis worker.
    • otherwise deletes it: DynamoDB with a conditional delete, S3 after re-checking the ETag.
  • Add a lock_sweeper section to the global config, wired into the deploy worker. It is off unless enabled: true and runs in dry-run mode unless dry_run: false.
  • The workflow reads its config through an activity each run, so a restarted worker with new config applies on the next sweep.
  • Only the exact lock keys are touched. Never the state, the md5 digest item, or anything else under the prefix.

Impact

No behavior change unless lock_sweeper.enabled is set; a disabled section converts to the zero value so existing config is unaffected. When enabled in dry-run it only logs and emits metrics. Existing workflows and activities are unchanged. To roll out, add the lock_sweeper section in atlantis-private (needs a submodule bump) with owner_prefix set to the identity Atlantis's workers take locks as. It is required when enabled and has no default, so automatic deletes only ever apply to those locks, run in dry-run on staging first, then set dry_run: false. Rollback is removing the section or setting enabled: false.

Testing

  • go test passes for server/neptune/lyft/locksweeper, server/config/..., server/neptune/temporalworker/... and cmd/... in a clean clone of main at 74bb20f. The full ./server/neptune/... suite also passed before the owner-prefix check was added; after it I re-ran only the packages listed. New tests cover the decision rules, key parsing, the delete guards (non-lock keys, md5 item, changed ETag, failed DynamoDB condition), query injection, the workflow including dry-run and the owner check, and config defaults and validation.
  • Not verified: that the Temporal visibility store accepts the WorkflowType / atlantis_repository / atlantis_root query; that the Atlantis role can list, get and delete the lock objects and scan and delete from the lock table; and anything against real AWS or Temporal. The S3 delete cannot be conditional with the v1 SDK, so a lock replaced between the ETag check and the delete would still be removed.

A worker that is killed mid-run leaves the S3 .tflock object and DynamoDB lock item behind,
and every later plan for that root fails with a 412 / ConditionalCheckFailedException until
someone deletes them by hand.

Add a cron-scheduled Temporal workflow that lists the locks and deletes the ones that are
provably orphaned: older than a threshold, held by the Atlantis worker, a plan lock, and with no
open Terraform workflow for the root in Temporal. Apply locks and locks held by anyone else only
alert. It is off unless lock_sweeper.enabled is set and runs in dry-run mode by default.
The tests and a doc comment used real internal repo and root names. Replace them with example names; the sweeper only parses <prefix>/<root>/terraform.tfstate.tflock, so behavior is unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant